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Method for spatial up-scaling of video frames 

FIELD OF THE INVENTION 

The present invention relates to a method and device for spatial up-scaling of an 
5 original video frame comprising p rows and q colums of pixels, where p and q are integers. 
It relates to a computer program product comprising program instructions for 
implementing said up-scaling method. 

This invention is, for example, relevant for television receivers or for personal 
computers, which have to be able to display still images or sequence of images at different 
10 scales. 

BACKGROUND OF THE INVENTION 

The development of high-resolution displays requires the use of efficient methods for 
spatial up-scaling of still images or sequence of images. Conventional methods of up-scaling 

15 include duplicating pixels and lines, exploitation of bilinear interpolation or other averaging 
techniques. However, these techniques result in poor quality of up-scaled images due to 
appearance of rough contours. Even if separable polyphase up-conversion filters are used» the 
problem of jagged lines remains. 

Other up-scaling methods use a discrete wavelet transform in a way similar to 

20 wavelet-based compression algorithms. The idea is based on the fact that a forward wavelet 
transform of an original image results in a low-low LL subband, which comprises low- 
frequency information in both horizontal and vertical directions and which is a downscaled 
version by a factor of 2 of said original image. On the contrary, if an original image is 
considered as a low-low subband received after a forward wavelet transform, then said 

25 original image may be up-scaled by applying an inverse wavelet transform. But the high- 
frequency subbands (i.e. the high-low HL, low-high LH, and high-high HH subbands), 
corresponding to the low-low subband LL (i.e. the original image) have to be constracted in 
order to apply the inverse wavelet transform. 

The US patent n°6,377,280 proposes an up-scaling method comprising a step of 

30 constructing these virtual high-frequency subbands HL, LH, and HH. According to said 
method, the original image is forward wavelet transformed to obtain HLl, LHl, HHl 
subbands of a first decomposition level. Then, values of the wavelet coefficients from 
subbands HLl and LHl, are fetched to the virtual subbands HL and LH, respectively. 
Because the number of wavelet coefficients in subbands HLl or LHl is four times smaller 
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than in virtual subbands HL or LH, the rest of coefficients in HL and LH subbands are set to 
zero according to a predetermined pattern. This prior art method is based on the assumption 
that wavelet coefficients at different decomposition levels are very similar in both amplitude 
and sign. However, this is not always true and the relocation of coefficients from one 
5 subband of a predetermined level, e.g. HLl, into another subband of a level lower than said 
predetermined level, e.g. HL, does not always provide a high picture quality. Moreover, this 
up-scaling method is rather complex and requires quite heavy computational resources. 

SUMMARY OF THE INVENTION 
10 It is an object of the invention to propose an up-scaling method, which is less 

complex than the one of the prior art. 

To this end, the up-scaling method in accordance with the invention is characterized 
in that it comprises the steps of: 

high-pass filtering the original video frame, considered as a low-low spatial frequency 
15 subband, in horizontal, vertical, and both directions, to constmct high-low, low-high, and 
high-high virtual spatial frequency subbands comprising p rows and q colums of pixels, 
respectively, 

applying an inverse wavelet transform to the constructed subbands and to the original 
video frame so that an up-sampled version of the original image is obtained. 
20 As a consequence, the generated virtual spatial frequency subbands have the same 

size as the original video frame. Thus, the up-scaling method in accordance with the 
invention does not need an additional step of combining a virtual spatial frequency subband 
of a first decomposition level, having size p/2*q/2, with null coefficients in order to obtain a 
virtual spatial frequency subband comprising p rows and q colums of data, as done in the 
25 prior art method. 

Moreover, by a proper choice of the high-pass filters, the picture quality is improved. 
This is for example the case if the high-pass filter is chosen among the same wavelet filters 
family than the filters used for the inverse wavelet transform. 

These and other aspects of the invention will be apparent from and will be elucidated 
30 with reference to the embodiments described hereinafter. 



BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will now be described in more detail, by way of example, with 
reference to the accompanying drawing, wherein: 
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- Fig. 1 is a block diagram of an up*scaling method in accordance with the invention, 

- Fig. 2 is a block diagram of a conventional two-dimensional inverse wavelet transform, 

- Fig. 3 A is a block diagram of a conventional lifting scheme, and 

- Fig. 3B is a block diagram of a simplified lifting scheme. 

DETAILED DESCRIPTION OF THE INVENTION 

The present invention relates to a method and device for spatial up-scaling of still 
images or of sequences of video images. 

The invention is based on the application of an inverse discrete wavelet transform 
(IWT) to the original image, considering said image as a low-low LL subband, and to the 
corresponding high-frequency subbands, which are efficiently predicted based on the original 
image information. The ability of a discrete wavelet transform to perform high quality 
approximation of edge features of an image makes it ideal for up-sampling applications. 

Figure 1 illustrates the general principle of the up-scaling method in accordance with 
the invention. 

At the first step of said method, the original image ORI, which comprises p rows and 
q columns of pixels, is considered as a virtual low-low LL subband received after a discrete 
forward wavelet transform of a virtual up-scaled image. 

After that, the high-frequency spatial subbands (i.e. low-high LH, high-low HL, and 
high-high HH) are constructed from the original image considered as the low-low LL 
subband using a high-pass filtering HF. The low-high LH subband contains information 
about horizontal edges in the original image; the high-low HL subband contains information 
about vertical edges, and the high-high HH subband contains information about diagonal 
edges. 

At a fmal stage, the proposed up-scaling method comprises a two-dimensional 
discrete inverse wavelet transform IWT, which is applied to the original image and to the 
constructed high-frequency subbands in order to obtain and transmit an up-scaled image UPI, 
with the number of rows and columns of pixels twice larger than in the original image, i.e. 2p 
rows and 2q columns of pixels. 

Fig. 2 illustrates said two-dimensional inverse wavelet transform. Said inverse 
wavelet transform comprises a first step UP2v of up-sampling by 2 along a vertical y- 
direction the different subbands LL, LH, HL and HH. Then, it comprises a step LPv of low- 
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pass filtering the up-sampled LL and HL subbands using a low-pass filter LP in a vertical 
direction. It also comprises a step HPv of high-pass filtering the up-sampled LH and HH 
subbands using a high-pass filter HP in a vertical direction. Then, the low-pass-filtered and 
up-sampled LL subband and the high-pass-filtered and up-sampled LH subband are added, 

5 resulting in an intermediate low-fi:equency ^ramc IL comprising 2p*q pixels . The low-pass- 
filtered and up-sampled HL subband and the high-pass-filtered and up-sampled HH subband 
are also added, resulting in an intermediate high-firequency frame IH comprising 2p*q pixels. 

The inverse wavelet transform comprises a second step UP2h of up-sampling by 2 
along a horizontal x-direction the intermediate frames IL and IH. Then, it comprises a step 

10 LPh of low-pass filtering the up-sampled IL frame using the low-pass filter LP in a horizontal 
direction. It also comprises a step HPh of high-pass filtering the up-sampled IH fi-ame using 
the high-pass filter HP in a horizontal direction. Then, the low-pass-filtered and up-sampled 
IL firame and the high-pass-filtered and up-sampled IH firame are added, resulting in the up- 
scaled image UPI comprising 2p*2q pixels. 

15 As an example, the low-pass filter is LPl = 1/2 [1, 1] and the high-pass filter is HPl = 

1/2 [1, -1], In other words, when the low-pass filter LP is applied in the horizontal x-direction 
to a pixel m, we have: 

LPl(m)=(x(m) + x(m+l))/2, 

and when applied in the vertical y-direction to a pixel n, we have: 
20 LPl(n)=(y(n) + y(n+l))/2. 

In the same manner, when the high-pass filter HP is applied in the horizontal x- 
direction to a pixel m, we have: 

HPl(m)=(x(m) - x(m+l))/2 

and when applied in the vertical y-direction to a pixel n, we have: 
25 HPl(n)=(y(n) - y(n+l))/2. 

It will be apparent to a person skilled in the art that the present invention is not 
limited to this pair of filters and that other pairs of filters are applicable, such as for example, 
LP2 = [0.02674875967204570800;-0.01686411909759044600; -0.07822325080633163500; 
0-266864091 15791321000; 0.60294902324676514000; 0.266864091 15791321000; 
30 -0.07822325080633163500; -0.01686411909759044600; 0.02674875967204570800] and 
HP2 = [0.045635882765054703, -0.028771763667464256, -0.2956358790397644, 
0.5574351615905762, -0.2956358790397644, -0.028771763667464256, 
0.045635882765054703] proposed by Antonini et al. in the paper entitled "Image Coding 
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Using Wavelet Transform" IEEE Trans, Image Processing, vol. 1, no. 2, pp. 205-220, April 
1992. 

The present invention proposes to construct coefficients of the virtual high-frequency 
5 subbands HL, LH and HH from the low-low LL subband using a high-pass filter. Said high- 
pass filter HP is applied to the original frame, i.e. the LL subband, in the horizontal direction, 
in the vertical direction and in both directions, in order to obtain the HL, LH, and HH 
subbands, respectively. 

According to an embodiment of the invention, the high-pass filter HP is chosen 
10 among the same wavelet filters family than the filters used for the inverse wavelet transform. 
This provides an almost optimal combination with the inverse wavelet transform. 

As an example, the high-pass filter HP used for the constraction step is the same as 
the high-pass filter HPf of a forward wavelet transform corresponding to the inverse wavelet 
transform used in the up-scaling method in accordance with the invention. More precisely, if 
1 5 LPf and HPf are the low-pass and high-pass filters of the forward wavelet transform, and LPi 
and HPi are the low-pass and high-pass filters of the inverse wavelet transform, then their 
relationships in the frequency domain are the following: 

LPi(©) = HPf(co-hi) 

HPi((D) = - LPf][(D+7t), where co is the frequency. 
20 Their relationships in the spatial domain is: 

HPf(k) = -(.l)^XPi(k) 

HPi(k) = (-l)^.LPf(k) where k is an integer comprised between - K and K, K having a 
predetermined value. 

For example, if the inverse wavelet transform filters are LPi = 1/4 [1, 2, 1] and HPi = 
25 1/4 [1, 2, -6, 2, 1], then the high-pass filter used for the construction of the HL, LH, and HH 
subbands is HP = HPf = 1/4 [1, -2, 1]. 

Thus, the proposed method does not require a complete forward wavelet transform 
but only a simplified version of said wavelet transform. At the same time, it allows a better 
reflection of high-frequency information of the original image because it does not exploit 
30 operation of down-sampling required for forward wavelet transform or does not copy 

information from subbands of a predetermined level into subbands of a level lower than said 
predetermined level. The simplified wavelet transform used for the subbgmd prediction 
involves only a high-pass filtering in one or both directions, without low-pass filtering and 
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down-sampling of wavelet coefficients, which are otherwise required by a conventional 
forward wavelet transform. 

Each of the high-frequency subbands is constructed by applying this high-pass filter 
to the low-low LL subband, i.e. the original image, in a horizontal direction, in a vertical 
5 direction, or in both directions. In order to receive the low-high LH subband, the original 
image is high-pass filtered in the vertical direction; thus horizontal edges are preserved. The 
high-low HL subband is constructed by high-pass filtering the original image in the 
horizontal direction. The high-high HH subband is constructed by applying the high-pass 
filter in both horizontal and vertical directions.Altematively, this high-high HH subband is 
10 constructed by applying a null filter to the original image, resulting in a HH subband filled 
with zeros. Such an alternative solution enables to save computational resources. The result 
of the high-pass filtering is that the size of the constructed subbands is then equal to the size 
of the original image. 

According to an embodiment of the invention, the step of constructing the LH, HL 
15 and HH subbands is implemented using a simplified lifting scheme. The conventional lifting 
scheme of a one-dimensional forward wavelet transform is depicted on Fig. 3 A. According to 
said scheme, an input signal x contained in the original image is split into even xe[n] and odd 
xo[n] samples. During a prediction phase, the high-frequency wavelet coefficients d[n] are 
computed as follows: 
20 d[n] = xo[n] - P(xe[n]), where PQ is a prediction fiinction. 

During an update phase, the low-frequency wavelet coefficients c[n] are computed as 
follows: c[n] = xe[n] + U(d[n]), where U() is an update fimction. Resolutions of c[n] and d[n] 
are twice smaller than the one of x[n] due to the operation of odd/even split. 

Because the up-scaling method does not implement a complete forward wavelet 
25 transform and the input signal x[n] ahready represents the low-frequency coefficients c[n], 
said up-scaling method is adapted to calculate the high-frequency wavelet coefficients d[n] 
and to normalize the low-frequency wavelet coefficients c[n]. Therefore, the operation of 
update UO is not required. Besides, the up-scaling should not implement the spUting of the 
input signal into sequences of even and odd samples, because the high-frequency wavelet 
30 coefficients d[n] are delivered with the same resolution as the input signal. The proposed 
simplified liffing scheme is depicted in Fig. 3B. According to said scheme, input samples 
x(n) are shifted, resulting in shifted samples xs(n). The high-frequency wavelet coefficients 
d[n] are computed on the basis of the input and shifted samples thanks to the prediction 
fiinction as follows: d[n] = ko.(xs[n] - P(x[n])), whereas the low-frequency wavelet 
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coefficients c[n] are derived from the input samples as follows: c[n] = ke.x [n], where ke and 
ko are normalization factors. 

It would be apparent to a person skilled in the art that the high-pass filter of the 
construction step can be derived from the filters of the inverse wavelet transform according to 
5 other different manners. 

According to an embodiment of the invention, pixel values of the original image are 
normalized by a normalization factor, said normalization factor depending on coefficients (or 
taps) of the high-pass filter, which is chosen for prediction of subbands and for inverse 
wavelet transform IWT. 

10 This normalization is required because the forward wavelet transform results in a low- 

low LL subband with coefficients having a different intensity value range than the one of a 
natural image. This intensity value range difference depends on the type of wavelet filters 
used. Thus, the normalization factor has to be defined based on the wavelet filters used for 
inverse wavelet transform. For example, if a 9/7 biorthogonal wavelet transform is used, then 

15 the value of the normalization factor is equal to the square of the sum of the low-pass filter 
coefficients. All pixels of the input fi:ame, considered as a LL subband, have to be multiplyed 
by this normalization factor. 

According to another embodiment of the invention, the constmction step and the 
inverse wavelet transform step are iterated until a predetermined up-scaling factor is reached. 

20 Said up-scaling factor can thus vary from 2 until 2^ where N is an integer strictly higher than 
one. 

The subbands low-high LH, high-low HL and high-high HH thus constructed contain 
direction dependent high-frequency information, which is predicted for up-scaling of the 
original image. Availability of this information will reduce the staircase artifacts, i.e. the so- 
25 called jagged lines, in the up-scaled image, said artifacts being typical for conventional up- 
sampling techniques. 

The proposed invention finds its application in spatial up-scaling of still images or of 
sequence of video frames decoded by wavelet-based decoders. For example, a spatially 
30 scalable stream compressed by a wavelet-based coder may be divided into several layers, 
each of which providing different resolution level. These layers may comprise a base layer 
containing the down-scaled version of the image, i.e. the LL subband, while the enhancement 
layers provide data required for reconstruction of the image at higher resolutions, i.e. the HL, 
LH, HH subbands. In case the enhancement layers are not available at the decoder side, i.e. 
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only the base layer is received by the decoder, then the original image may be reconstructed 
from the down-scaled version of the image decoded from the base layer using an up-scaling 
device implementing the up-scaling method in accordance with the invention. 

According to an embodiment of the invention, the proposed spatial up-scaling device 
5 is incorporated into the wavelet-based decoder. Thus, it does not require any additional 
dedicated architecture blocks, as inverse wavelet transform is akeady utilized for the image 
decoding. Therefore prediction of the high-frequency subbands may be implemented without 
additional overhead. 

It will be apparent to a person skilled in the art that the up-scaling device can also be 
10 incorporated into a displaying apparatus receiving the decoded video frames. The displaying 
apparatus is, for example, a television receiver or a personal computer. 

The above-described application should not limit the scope of the invention. The 
proposed up-scaling method may also be used independently of wavelet-based encoding / 
decoding systems. 

15 

The up-scaling method in accordance with the invention can be implemented by 
means of items of hardware or software, or both. Said hardware or software items can be 
implemented in several manners, such as by means of wired electronic circuits or by means 
of an integrated circuit that is suitable programmed, respectively. The integrated circuit can 

20 be contained in a decoder, in a personal computer or in a television receiver for example. A 
set of instructions contained, for example, in a memory may cause the integrated circuit to 
carry out the different steps of the up-scaling method. The set of instructions may be loaded 
into the memory by reading a data carrier such as, for example, a disk. A service provider can 
also make the set of instructions available via a communication network such as, for 

25 example, the Internet. 

Any reference sign in the following claims should not be construed as limiting the 
claim. It will be obvious that the use of the verb "to comprise" and its conjugations do not 
exclude the presence of any other steps or elements besides those defined in any claim. The 
30 word "a" or *W preceding an element or step does not exclude the presence of a plurality of 
such elements or steps. 



